This was an exercise in leveraging the machine learning library Tensorflow.JS in conjunction Mediapipe's Hands computer vision model to emulate spatial computing interactions, such as those in VR and LeapMotion, in the browser using only standard webcam RGB video input.

Conceptually, this was inspired by my experience with cadaver dissections, as well as the early popularity of LeapMotion sensors in the medical field, to avoid having to physically touch an object (textbook, screen, mouse, etc.) to interact with it.

The final application allows users to navigate using gestures from both hands, in both 2D and 3D space. A clipping plane function allows users to create a cross-sectional view of the brain in the saggital, coronal, and transverse planes, while the MRI view is available on the right side and corresponds to the location of the clipping planes. It is intended for desktop use (ideally 16in+ display), where the user is positioned approximately 1m from the webcam.

Users can rotate the model with a closed wiping hand gesture, and select structures by 'touching' them with their index finger. The model will then highlight the selected structure, and display the name of the structure in the centre box in the bottom grid. Doing so will also drop a 'pin' in the area of selection, which is also represented in the MRI view; this is meant to be leverage anatomy lab conventions for marking a structure to help indicate the selection in the MRI. The bottom grid provides contextual information for the structures that immediately surround the selected structure.

A horned hands gesture deselects all structures.

Using T1-weighted MRI data from the Human Connectome Project, I created a 3D model of the brain using Blender. I then used ZBrush to create a high-resolution texture map for the model.

To allow for a high degree of granularity when selecting structures, the model was split into 120 meshes based on DKT cortical regions, white matter, and subcortical structures. All sections are solid and manifold.